Papers with LM fine-tuning
To Clarify or not to Clarify: A Comparative Analysis of Clarification Classification with Fine-Tuning, Prompt Tuning, and Prompt Engineering (2024.naacl-srw)
Copied to clipboard
| Challenge: | Xu et al., 2019) show that pre-trained language model fine-tuning and prompt tuning are better than manual prompt engineering for clarification identification. |
| Approach: | They propose to use pre-trained language model fine-tuning, prompt tuning and manual prompt engineering to model clarification identification. |
| Outcome: | The proposed model outperforms pre-trained language model fine-tuning, prompt tuning and manual prompt engineering on the task of clarification identification. |
Self-Influence Guided Data Reweighting for Language Model Pre-training (2023.emnlp-main)
Copied to clipboard
| Challenge: | Language Models (LMs) pre-trained with selfsupervision on large text data are the default starting point for developing models for various downstream tasks. |
| Approach: | They propose a method for jointly reweighting samples by leveraging self-influence scores as an indicator of sample importance and pre-training. |
| Outcome: | The proposed method promotes novelty and stability for model pre-training. |
Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies (2021.eacl-main)
Copied to clipboard
| Challenge: | Biomedical question-answering (QA) provides users with high-quality information from a vast scientific literature. |
| Approach: | They propose to use a biomedical entity-aware masking strategy to fine-tune masked language models to their domains. |
| Outcome: | The proposed approach is an adaptation process for masked LMs, not memory or components. |
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021.naacl-main)
Copied to clipboard
| Challenge: | PTLMs can exhibit biases against protected groups in a host of modeling tasks . but, fine-tuned LMs may propagate bias to downstream classifiers . |
| Approach: | They propose to use upstream bias mitigation techniques to reduce bias on downstream tasks by fine-tuning an upstream model and applying it to a downstream model. |
| Outcome: | The proposed model reduces bias on hate speech detection, toxicity detection and coreference resolution tasks over bias factors. |
Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a new algorithm to estimate fine-tuning performance for a target task is proposed . conventional subset selection methods require repeated training on subsets of auxiliary tasks . |
| Approach: | They propose an algorithm to fine-tune a language model for a target task by optimally using auxiliary tasks' information. |
| Outcome: | The proposed method can estimate fine-tuning performance on CPUs in seconds. |
TokenDrop + BucketSampler: Towards Efficient Padding-free Fine-tuning of Language Models (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-training of Language Models (LMs) is a challenge due to its huge computational footprint. |
| Approach: | They propose a framework that improves the efficiency and accuracy of LM fine-tuning by removing padding tokens from sequences that are variable-length . |
| Outcome: | The proposed framework accelerates fine-tuning on diverse downstream tasks by 10.61X while producing models that are up to 1.17% more accurate compared to conventional fine-uning. |